Performance features in clinical skills assessment: Linguistic and cultural factors in the Membership of the Royal College of General Practitioners examination

This book is based on research looking at performance in clinical skills assessment from a linguistic and cultural perspective, with a view to understanding why there are such differential pass rates and giving suggestions on how this issue can be tackled. It is both a research report and a guide to the sociolinguistic methodology used. While the findings are based on a research project in partnership with the Royal College of General Practitioners, they are applicable to many other medical settings where standardised examinations of simulated consultations are used. More widely, this research addresses a central paradox in institutional life – how to balance validity in assessments and be fair to a diverse group of candidates in an increasingly diverse society, while maintaining reliability with standardised and universal marking criteria. It has been widely acknowledged that candidates from overseas fair less well in such examinations. A close look at the interactions which make up these simulated consultations shows that there are complex and subtle differences between passing and failing candidates which cannot be explained simply as ‘language’ and ‘cultural’ differences and put in a box separate from issues of fairness. These structured examinations, unintentionally, contribute to the weight of the assessment on overseas candidates, particularly in how interpersonal effectiveness is judged both explicitly and implicitly. The research has identified a range of successful candidate strategies which form the basis of a set of e–learning materials to be published by the RCGP. It also suggests that aspects of the exam, notably the more subjective features of interpersonal skills, are not best assessed in highly structured exams. This area needs to be better defined, using a new analytic language, to debate how and where it could be most effectively and fairly assessed

Sociolinguistic factors affecting performance in the Clinical Skills Assessment of the MRCGP: a mixed-methods approach

Background Differential performance in clinical skills assessments is a widespread phenomenon, for which there remain few explanations. Aim To better understand the conversational contexts of simulated consultations and how candidates actually behave in these consultations and to determine sociolinguistic factors for high- and low-performing candidates. Design & setting Taking the Membership of the Royal College of General Practitioners' (MRCGP) clinical skills assessment (CSA) examination as a model, this research applied sociolinguistic analyses to case videos of 198 consecutive candidates presenting for the CSA examination. Method Using a mixed-methods approach, both quantitative and qualitative sociolinguistics methodologies were combined to analyse video consultations, and findings were compared with those from group discussions with MRCGP examiners. Results There is more ‘talk’ in simulated consultations than in real life. On macroanalysis, there was little difference between poor- and well-performing candidates. However, microanalysis found subtle differences in structuring consultations, metacommunication, picking up cues, and misunderstandings with and giving explanations to patients. Formulaic talk, contrary to examiners’ perceptions was more common in successful candidates, but it was personalised and sited appropriately in the consultation. Conclusion This is an interactionally demanding form of clinical assessment, that requires giving support to candidates and a more analytic approach to the development of interpersonal skills. Sociolinguistic features of consulting to help trainers and candidates prepare for the CSA are identified

Involving lay and professional stakeholders in the development of a research intervention for the DEPICTED Study

Aim: This paper focuses on stakeholders’ active involvement at key stages of the research as members of a Stakeholder Action Group (SAG), particularly in the context of lay stakeholder involvement. Some challenges that can arise and wider issues (e.g. empowerment, the impact of user involvement) are identified and explored within the literature on service user involvement in health care research, reflecting on the implications for researchers. Background: In the DEPICTED study, lay and professional stakeholders were actively involved in developing a complex research intervention. Lay stakeholders comprised teenage and adult patients with diabetes, parents and patient organization representatives. Professional stakeholders were from a range of disciplines. Methods: Three 1-day research meetings were attended by 13–17 lay stakeholders and 10–11 professional stakeholders (plus researchers). The SAG was responsible for reviewing evidence, advising on developing ideas for the research intervention and guiding plans for evaluation of the intervention in a subsequent trial. Formal evaluations were completed by stakeholders following each SAG meeting. Results: Throughout the first (developmental) stage of this two-stage study, lay and professional stakeholders participated or were actively involved in activities that provided data to inform the research intervention. Lay stakeholders identified the need for and contributed to the design of a patient-held tool, strongly influenced the detailed design and content of the research intervention and outcome questionnaire, thus making a major contribution to the trial design. Conclusion: Stakeholders, including teenagers, can be actively involved in designing a research intervention and impact significantly on study outcomes

The effect of the Talking Diabetes consulting skills intervention on glycaemic control and quality of life in children with type 1 diabetes: cluster randomised controlled trial (DEPICTED study)

Objective To evaluate the effectiveness on glycaemic control of a training programme in consultation skills for paediatric diabetes teams

Learning to mark: a qualitative study of the experiences and concerns of medical markers

BACKGROUND: Although there is published research on the methods markers use in marking various types of assessment, there is relatively little information on the processes markers use in approaching a marking exercise. This qualitative paper describes the preparation and experiences of general practice (GP) teachers who undertake marking a written assessment in an undergraduate medical course. METHODS: Semi-structured interviews were conducted with seven of the 16 GP tutors on an undergraduate course. The purposive sample comprised two new markers, two who had marked for a couple of years and three experienced markers. Each respondent was interviewed twice, once following a formative assessment of a written case study, and again after a summative assessment. All interviews were audio-taped and analysed for emerging themes. A respondent validation exercise was conducted with all 16 GP tutors. RESULTS: Markers had internal concerns about their ability to mark fairly and made considerable efforts to calibrate their marking. They needed guidance and coaching when marking for the first time and adopted a variety of marking styles, reaching a decision through a number of routes. Dealing with pass/fail borderline scripts and the consequences of the mark on the student were particular concerns. Even experienced markers felt the need to calibrate their marks both internally and externally CONCLUSION: Previous experience of marking appears to improve markers' confidence and is a factor in determining the role which markers adopt. Confidence can be improved by giving clear instructions, along with examples of marking. The authors propose that one method of providing this support and coaching could be by a process of peer review of a selection of papers prior to the main marking. New markers in particular would benefit from further guidance, however they are influenced by others early on in their marking career and course organisers should be mindful of this when arranging double marking

Development and Evaluation of a Psychosocial Intervention for Children and Teenagers Experiencing Diabetes (DEPICTED): a protocol for a cluster randomised controlled trial of the effectiveness of a communication skills training programme for healthcare professionals working with young people with type 1 diabetes

Background Diabetes is the third most common chronic condition in childhood and poor glycaemic control leads to serious short-term and life-limiting long-term complications. In addition to optimal medical management, it is widely recognised that psychosocial and educational factors play a key role in improving outcomes for young people with diabetes. Recent systematic reviews of psycho-educational interventions recognise the need for new methods to be developed in consultation with key stakeholders including patients, their families and the multidisciplinary diabetes healthcare team. Methods/design Following a development phase involving key stakeholders, a psychosocial intervention for use by paediatric diabetes staff and not requiring input from trained psychologists has been developed, incorporating a communication skills training programme for health professionals and a shared agenda-setting tool. The effectiveness of the intervention will be evaluated in a cluster-randomised controlled trial (RCT). The primary outcome, to be measured in children aged 4-15 years diagnosed with type 1 diabetes for at least one year, is the effect on glycaemic control (HbA1c) during the year after training of the healthcare team is completed. Secondary outcomes include quality of life for patients and carers and cost-effectiveness. Patient and carer preferences for service delivery will also be assessed. Twenty-six paediatric diabetes teams are participating in the trial, recruiting a total of 700 patients for evaluation of outcome measures. Half the participating teams will be randomised to receive the intervention at the beginning of the trial and remaining centres offered the training package at the end of the one year trial period. Discussion The primary aim of the trial is to determine whether a communication skills training intervention for specialist paediatric diabetes teams will improve clinical and psychological outcomes for young people with type 1 diabetes. Previous research indicates the effectiveness of specialist psychological interventions in achieving sustained improvements in glycaemic control. This trial will evaluate an intervention which does not require the involvement of trained psychologists, maximising the potential feasibility of delivery in a wider NHS context. Trial registration Current Controlled Trials ISRCTN61568050

Simulated consultations: a sociolinguistic perspective

Background: Assessment of consulting skills using simulated patients is widespread in medical education. Most research into such assessment is sited in a statistical paradigm that focuses on psychometric properties or replicability of such tests. Equally important, but less researched, is the question of how far consultations with simulated patients reflect real clinical encounters – for which sociolinguistics, defined as the study of language in its socio-cultural context, provides a helpful analytic lens. Discussion: In this debate article, we draw on a detailed empirical study of assessed role-plays, involving sociolinguistic analysis of talk in OSCE interactions. We consider critically the evidence for the simulated consultation (a) as a proxy for the real; (b) as performance; (c) as a context for assessing talk; and (d) as potentially disadvantaging candidates trained overseas. Talk is always a performance in context, especially in professional situations (such as the consultation) and institutional ones (the assessment of professional skills and competence). Candidates who can handle the social and linguistic complexities of the artificial context of assessed role-plays score highly – yet what is being assessed is not real professional communication, but the ability to voice a credible appearance of such communication. Summary: Fidelity may not be the primary objective of simulation for medical training, where it enables the practising of skills. However the linguistic problems and differences that arise from interacting in artificial settings are of considerable importance in assessment, where we must be sure that the exam construct adequately embodies the skills expected for real-life practice. The reproducibility of assessed simulations should not be confused with their validity. Sociolinguistic analysis of simulations in various professional contexts has identified evidence for the gap between real interactions and assessed role-plays. The contextual conditions of the simulated consultation both expect and reward a particular interactional style. Whilst simulation undoubtedly has a place in formative learning for professional communication, the simulated consultation may distort assessment of professional communication These sociolinguistic findings contribute to the on-going critique of simulations in high-stakes assessments and indicate that further research, which steps outside psychometric approaches, is necessary